Indoor identification of individuals through footstep induced structural vibration

ABSTRACT

This invention introduces an indoor person identification system that utilizes the capture and analysis of footstep induced structural vibrations. The system senses floor vibration and detects the signal induced by footsteps. Then the system then extracts features from the signal that represent characteristics of each person&#39;s unique gait pattern. With these extracted features, the system conducts hierarchical classification at an individual step level and at a collection of consecutive steps level, achieving high degree of accuracy in the identification of individuals.

RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/176,108, filed Feb. 9, 2015.

GOVERNMENT RIGHTS IN THE INVENTION

This invention was made with government support under National ScienceFoundation No. CNS-1149611. The government has certain rights in thisinvention.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a national phase claiming the benefit of andpriority to International Patent Application No. PCT/US 2016/017186,entitled “INDOOR IDENTIFICATION OF INDIVIDUALS THROUGH FOOTSTEP INDUCEDSTRUCTURAL VIBRATION,” filed Feb. 9, 2016, which claims priority toProvisional U.S. Patent Application No. 62/176,108 filed Feb. 9, 2015,which are hereby incorporated by reference in their entireties.

BACKGROUND OF THE INVENTION

Many smart building applications require indoor identification ofindividuals for personalized tracking and monitoring services. Forexample, in a nursing home, identifying monitored patients and trackingindividual activity range helps nurses understand the condition ofpatients. Similarly, such identification information can also be used insmart stores/malls to analyze shopping patterns of customers.

Various methods and apparatuses have been explored for identification ofindividuals. These methods and apparatuses utilize biometrics (face,iris, fingerprints, hand geometry, gait, etc.) and sensing technologies(vision, sound, force, etc.). Some biometrics, such as iris,fingerprints and hand geometry achieve relatively high identificationaccuracy and are widely used for access control. However, they oftenrequire human interactions, and, as such, they have limited usefulnessfor ubiquitous smart building applications. With other methods, such asfacial and gait recognition, it is often difficult to get enough sensingresolution required for recognition from a distance, particularly whenused in surveillance applications. Numerous sensing technologies havebeen explored and proven useful and efficient, but all have limitations.Vision-based methods often require line-of-sight, with performancedependent upon lighting conditions, and may require high computationalcosts, which limits their viability. Likewise, sound-based methods havelimitations when deployed in conversation sensitive areas, as they areprone to be affected by ambient audio. Force-based methods typicallyutilize specialized floor tile sensors for footstep detection, resultingis the requirement for dense deployment at a high installation cost.

SUMMARY OF THE INVENTION

This invention performs identification of individuals via footstepinduced structural vibration analysis. People walk differently, andtherefore their footsteps result in unique structural vibrations. Theinvention measures these vibrations, detects signals induced byfootsteps, extracts features from these signals, and applies ahierarchical classifier to these features to identify each registereduser with a high confidence level.

Due to better wave attenuation properties in solids, with properamplification, the invention can detect individuals at a relativelylarge range. As a result, the invention has a sensing density that islow compared to known force-based methods. Compared to vision-based andsound-based methods, the invention measurement suffers less interferencefrom obstacles that move around, because the vibrations travel in thestructure itself. Furthermore, the installation of the invention isnon-intrusive, consisting of one or more geophones installed on or nearthe floor surface, which can be accomplished without alteration thestructure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows recorded step events from three different people wearingsoft-soled shoes, with each row showing a different person. The leftcolumn shows the time domain of the step event, while the right columnshows the frequency domain of the same step event.

FIG. 2 shows three separate recordings of step events from the sameperson. The left column shows the time domain of the step event, whilethe right column shows the frequency domain of the same step event.

FIG. 3 presents the schematic overview of the components of theinvention, highlighting the required functional modules.

DETAILED DESCRIPTION OF THE INVENTION

Each person has a unique walking pattern due to many factors, including,for example, individual physical characteristics, the center of gravityposition during the walk, the way feet contact the ground, etc. Due toeach person's unique walking pattern, there is a uniqueness andconsistency of the footstep induced floor vibration for each person.

The floor vibration signal induced by a footstep is referred to hereinas a step event. A sequence of step events from a continuous walk isreferred to herein as a trace.

The floor vibration signal is captured by one or more sensing modules,each of which consists of three major parts: a geo-phone, an amplifier,and an analog-to-digital converter. The geophone is set on the floor ofthe structure to capture floor vibration signals. The analog signal isthen amplified. In the preferred embodiment, the amplification isperformed by connecting the geophone to an op-amp with an empiricalamplification gain of approximately 1000, which allows approximately asensing range of about 10 m for particular factors including floor type,shoe type, etc. However, as would be realized by one of skill in theart, many methods of amplification could be used. A sampling rate of 25kHz allows the capture of a wide frequency range of signalcharacteristics, but other sampling rates could be used.

In tests of the system using this sensor module, step events fromdifferent people were recorded, showing distinguishable variations inboth time and frequency domains. FIG. 1 shows step events from threepeople, labelled (a), (b) and (c). The left and right columns showcorresponding time and frequency domain signals from the same step eventfor each person, respectively. In addition, dotted lines anddashed/dotted lines indicate locations of peaks and valleys in thefrequency domain, respectively. As shown in FIG. 1, the locations ofpeaks and valleys vary among different people, which can be used asfeatures to identify them.

Step events from one person bear resemblance between each other. FIG. 2shows three step events from one trace (i.e., from a series of steps bythe same person). The left and right columns show corresponding time andfrequency domain signals, respectively. Note that the step events spansimilar time duration with nearly identical velocity profiles in thetime domain. The frequency domain patterns are well aligned across thethree step events. This data demonstrates that a step event is afeasible metric for identification of individuals.

As shown in FIG. 3, the identification system of the present inventioncontains three modules: sensing 10, footstep analysis 20, anddecision-making 30. FIG. 3 displays the relations of these modules.

Sensing module 10 (described above) performs floor vibration sensing 12.The vibrations sensed are those that are induced by a person walkingacross a floor surface Sensing hardware 10 amplifies the signal receivedfrom the sensor and outputs a digital signal derived from the amplifiedanalog output of the sensor. As discussed above, in a preferredembodiment of the invention, the sensor is a geophone of a type that iswell known and commercially available, however, other types of sensorsmay be used. The system may use multiple sensing modules 10, dependingupon the desired area of coverage.

Footstep analysis module 20 takes a trace of step events and extractsindividual step events therefrom. Features representing characteristicsof each step event are then extracted.

The key to identification of individuals is to extract and analyze thecharacteristics of step events. There are two major components in thefootstep analysis module. The first is step extraction 22 to obtain stepevents, and the second is feature extraction 24, which characterizesstep events via feature extraction.

Step events contain a person's identity information, while the intervalbetween step events is mainly noise. Therefore, to identify people, stepevents need to be extracted from the trace containing the entirevibration signal by step extraction module 22. The noise is modeled as aGaussian distribution, and then an anomaly detection method is used toextract step events. The threshold value to detect a step event isdetermined by an allowable false alarm rate.

Two detection algorithms have been developed for extracting step eventsfrom the trace containing the entire vibration signal. The firstdetection algorithm is threshold-based method and uses the timerepresentation of the signal. This method finds the threshold using thebackground noise distribution and a footstep event is indicated wheneverthe energy of signal exceeds a defined threshold. The second detectionalgorithm uses the time-frequency representation of the signal. Thisapproach is able to deal with signals with very low signal-to-noiseratio where it is difficult to differentiate between the backgroundnoise and footstep-induced vibrations and improves the accuracy bydistinguishing between footsteps and other sources of non-stationaryexcitation. Some examples of such sources include vibrations induced bydropping an object and shutting a door. This algorithm uses thecharacteristics of structure to find the frequency components of thesignal which are more robust to background noise. Furthermore, itincludes a classification algorithm which distinguishes betweenfootstep-induced vibrations and vibrations induced by non-stationarysignals.

Feature extraction module 24, extracts features from selected stepevents. The events from which to the features are extracted are selectedbased on their signal to noise ratio. Features can be more efficientlyextracted from step events in a trace having a high signal-to-noiseratio. Features of the selected steps are then extracted to characterizethe footsteps.

Step events in one trace may have different signal-to-noise ratiosdepending on the relative distance of the location of each step event toa sensor. This leads to a variation in classification performance. Asmall number of step events closest to the sensor, and consequently withthe highest signal-to-noise ratio, are selected for classification.

Once the step events are selected, they are normalized to remove effectsof the distance between the footstep location and the sensor, and fordifferent types of floor surfaces, for example, a hard floor versus acarpeted floor. Step events closer to the sensor have a higher signalenergy, which is calculated as the sum of squared signal values. Eachselected step event is divided by its signal energy to normalize fordifferences in the distance of each step event from the sensor, therebyremoving the distance effect, the distance of each step event from thesensor is irrelevant to characterizing the step event for a particularperson and contains no identify information.

After normalization, features are computed in both time and frequencydomains to present different characteristics of step events for eachperson. Time domain features may include, but are not limited tostandard deviation, entropy, peak values, partial signal before andafter the maximum peak, etc. In the frequency domain, features mayinclude, but are not limited to spectrum centroid, locations andamplitudes of peaks, power spectrum density, etc.

Once these features are extracted, decision-making module 30 takes thefeatures and runs through a hierarchical classifier, which includes bothstep level classification 32 and trace level classification 34. Theidentification of individuals is modeled as a hierarchicalclassification problem in the invention. A hierarchical classifierincludes step level classifications 32 and trace level classifications34. Identification accuracy is increased by utilizing the fact thatsteps from the same trace belong to the same person. The classified stepevents are compared against a database 36 of previous step events fromidentified individuals to accurately identify the individual.

The system takes features of step events from different people's tracesto generate a classification model using a Support Vector Machine, whichmaximizes the distance between data points and the separatinghyper-plane. The step level classification 32 returns both theidentification label and the confidence level from testing the stepevent.

By classifying identity at trace level 34, classification uncertainty isreduced by eliminating outlier step events from the step levelclassification 32, thereby enhancing the overall identification accuracyof the system.

Each step event classified obtains an identification label and aconfidence level as the result of the step level classification 32.Since multiple steps events with the highest signal-to-ratio arereferenced a confidence matrix P_(s×n) is created, where n is the numberof people to be classified, and s is the number of step events selectedfrom the trace. The identity of the step event with highest confidencelevel is selected to be the identity of the entire trace.

Achieving high accuracy for the classified step events is important.When a new person's trace is detected, it is possible that step eventsin the new trace are not similar to any of the footsteps in database 36.In this case, the confidence levels of all steps in a trace are equallylow, and the system detects such situations. The confidence levelthreshold CL_(threshold) is set to determine a reliable classificationresult. The trace is considered to be identifiable when the confidencelevel is higher than the confidence level threshold. Otherwise, thetrace is determined to be unclassifiable (i.e., the trace of apreviously un-identified person). The system can adjust this thresholdto obtain different identification accuracy based on the application.

In tests of the system, various numbers of persons, and various types ofstructures were used, and the system was found to provide a highidentification accuracy.

Many applications of the system have been identified in the areas ofindividual monitoring, analysis of group behavior and security.

Individual identification and monitoring can be used to detect childrenor elderly patients in an in-home setting, where the system can detectand identify individuals and respond accordingly, for example, if theyappear alone in designated area (e.g., the kitchen or bathroom), or ifthey leave the premises. For elderly subjects, the system can be used toanalyze walking patterns to predict fatigue level, which may be usefulin and prevent fall events from occurring. Finally, individualidentification can be used to identify individuals in a smart space, andpersonalize the environmental settings, for example, by detecting theidentity of an individual as they walk through the front door, the smartsystem can start their computer before their arrival, then, by trackingthe individual to the elevator, the smart system can play their favoritesongs in the elevator. Likewise, the system could also set customizedtemperature, turn on lights, unlock doors, etc.

The system may also be applied to monitor and analyze group behavior. Ina supermarket, shopping mall or airport environment, the system mayrecognize individual shopping patterns and understand the group shoppingpattern based on the characterization from the footstep inducedvibration signals (e.g., height, weight, gender, etc.). In a smartoffice type environment, the system could recognize the activity rangeof each individual and assign resources/space and manage energyconsumption based on the optimized convenience.

Lastly, there are security applications for the system. For example, thesystem may be used to authorize access to a particular area bydetermining if the detected footsteps fit the profile of an authorizedindividual. The system may also be useful in theft detection, bydetecting changes in the pattern of individual footsteps due to hiddenobjects on the body of the individual. Lastly, the system may be able todetect specific gait patterns due to individuals carrying weapons ontheir body.

Although the invention is illustrated and described herein withreference to specific embodiments, the invention is not intended to belimiting to the details shown. Rather, various modifications may be madein the details without departing from the invention.

We claim:
 1. A system for identifying individuals in a structurecomprising: a vibration sensor, disposed on a walking surface in thestructure; and a processor, in communication with the vibration sensor,the processor running software performing the functions of: receiving,from the vibration sensor, a signal representing vibrations generated byone or more footsteps of the individual; extracting, from the signal,one or more discrete step events, each step event representing onefootstep of the individual; extracting, from each of the one or morediscrete step events, one or more features, the one or more featuresincluding both time domain features and frequency domain features; andidentifying the individual, the identification being based wholly orpartially on an analysis of the one or more features.
 2. The system ofclaim 1 wherein the vibration sensor is a geophone positioned on asurface where the individual is walking.
 3. The system of claim 1wherein the signal from the vibration sensor is amplified before beingcommunicated to the processor.
 4. The system of claim 1 wherein thesignal from the vibration sensor is digitized before being communicatedto the processor.
 5. The system of claim 1 wherein the discrete stepevents are extracted by: modeling intervals in the signal betweendiscrete footsteps as noise modelled as a Gaussian distribution; andusing an anomaly detection algorithm to extract the discrete stepevents.
 6. The system of claim 1, the discrete step events extracted byan algorithm for detecting when the energy of the signal representingvibrations generated by the one or more footsteps of the individualexceeds a predefined threshold.
 7. The system of claim 1, the discretestep events extracted by an algorithm to detect frequency components ofthe signal representing vibrations generated by one or more footsteps ofthe individual which are more robust to background noise, furthercomprising: classifying the detected frequency components asfootstep-induced vibrations or vibrations induced by non-stationarysignals.
 8. The system of claim 1 wherein the features are extractedonly from the discrete step events having a signal to noise ratio abovea certain threshold.
 9. The system of claim 8 wherein the discrete stepevents are normalized to compensate for distance from the vibrationsensor of each footstep.
 10. The system of claim 8 wherein the discretestep events are normalized to compensate for different floor surfaces.11. The system of claim 1 wherein the time domain features are selectedfrom a group consisting of standard deviation, entropy, peak values andpartial signals before and after maximum peak values.
 12. The system ofclaim 1 wherein the frequency domain features are selected from a groupconsisting of: spectrum centroid, locations and amplitudes of peaks andpower spectrum density.
 13. The system of claim 1 further comprising adatabase containing models of previously identified individuals, themodels consisting of features previously extracted from discretefootstep signals of the individuals.
 14. The system of claim 13 whereinthe identification is based on a comparison of the features with thepreviously extracted features in the model stored in the database. 15.The system of claim 14 wherein the identification is based partially onan analysis of features extracted from the signal representingvibrations generated by one or more footsteps.
 16. The system of claim 1wherein signals representing vibrations generated by one or morefootsteps of the individual are collected from two or more vibrationsensors.
 17. The system of claim 1 wherein identifying the individualfurther comprises: analyzing the one or more features using ahierarchical classifier to classify each discrete step event; assigningan identity of an individual and a confidence level to each step event;and identifying an individual as the individual associated with the stepevent having the highest confidence level.
 18. The system of claim 17wherein the hierarchical classifier classifies the one or more discretestep events and a trace comprising multiple step events.
 19. Acomputer-implemented method for identifying an individual in a structurebased on vibrations generated by footsteps comprising the steps of:collecting a signal representing one or more footsteps of the individualfrom a vibration sensor disposed on a walking surface within thestructure; extracting, from the signal, one or more discrete stepevents, each step event representing one footstep of the individual;extracting, from each of the one or more discrete step events, one ormore features, the one or more features including both time domainfeatures and frequency domain features; and identifying the individual,the identification being based wholly or partially on an analysis of theone or more features.
 20. The method of claim 19 wherein theidentification is made by comparing the one or more features to a modelof the individual stored in a database.
 21. The method of claim 20further comprising the steps of selecting the discrete step eventshaving the highest signal to noise ratio; and extracting the featuresonly from the selected discrete step events.
 22. The system of claim 21further comprising the step of normalizing the selected discrete stepevents to compensate for distance from the vibration sensor of eachfootstep prior to extracting the features from the discrete step events.23. The method of claim 19 wherein the identifying step includes ananalysis of features extracted from the signal representing vibrationsgenerated by one or more footsteps.
 24. The method of claim 19 whereinidentifying the individual further comprises: analyzing the one or morefeatures using a hierarchical classifier to classify each discrete stepevent; assigning an identity of an individual and a confidence level toeach step event; and identifying an individual as the individualassociated with the step event having the highest confidence level. 25.The method of claim 24 wherein the hierarchical classifier classifiesthe one or more discrete step events and a trace comprising multiplestep events.